Microsoft Word - CIKM_relevanceModel_V01_ACMformat

Lack of high quality relevance labels is a common challenge in the early stage of search engine development. In media search, due to the high recruiting and training cost, the labeling process is usually conducted by a small number of human judges. Consequently, the generated labels are often limited and biased. On the contrary, the click data that is extracted from a large population of real users is massive and less biased. However, the click data also contains considerable noise. Therefore, more and more researchers have begun to focus on combining those two resources to generate a better ground-truth approximation. In this paper, we present a novel method of generating the relevance labels for media search. The method is based on a generative model that considers human judgment, position, and click status as observations generated from a hidden relevance with multinomial prior. The model considers the position bias with a requirement that the click status depends on both the hidden relevance and the position. We infer the model parameters by using a Gibbs sampling procedure with hyperparameter optimization. From experiments on the Xbox’s data, the newly inferred relevance labels significantly increase the data volume for ranker training and have demonstrated superior performance compared to using the limited human labels only, the click-through-rates only, and the heuristic combination of the two.